Abstract 3 Title Page 


Title: Core Academic Language skills: Moving beyond vocabulary knowledge to predict reading 
comprehension 

Authors and Affiliations: 

Paola Uccelli, Harvard Graduate School of Education 

Emily Phillips Galloway, Harvard Graduate School of Education 

Ha Yeon Kim, Harvard Graduate School of Education 

Christopher D, Barr, Texas Institute for Measurement and Statistics (TIMES), University of 
Houston 


SREE Spring 2015 Conference Abstract Template 



Abstract 3 Body 


Background / Context: 

Despite a longstanding awareness of aeademic language as a pedagogieally-relevant researeh 
area, the construct of academic language proficiency— understood as a more comprehensive set 
of skills than just academic vocabulary— has remained only vaguely specified (Cummins, 1979, 
2008). Academic language proficiency, broadly understood as proficiency in “the language of 
schooling” (Schleppegrell, 2001, 2004), has increasingly become a topic of interest in 
educational circles because of its hypothesized contribution to reading comprehension and 
content-area achievement (Abedi & Herman, 2010; August and Shanahan, 2006; Bailey, 2007; 
Biancarosa & Snow, 2006). It has become commonplace to argue that the reading 
comprehension difficulties documented for a large proportion of students in grades 4 and above 
(particularly, for students living in poverty and/or acquiring English as a second language in the 
U.S.) are, in large part, the result of students’ challenges understanding the academic language of 
school texts. Moreover, students’ academic language proficiency is being increasingly 
understood in the field as a malleable factor that can be effectively scaffolded through high- 
quality instruction. One of the central shifts in practice advocated by the recently and widely- 
adopted Common Core State Standards in the U.S. calls for “regular practice with academic 
language and complex texts” throughout the upper elementary and secondary school years 
(National Governors Association, 2010). Paradoxically, though, an operational definition of 
academic language proficiency that would be sufficiently precise to inform instruction remains 
elusive. In the absence of a comprehensive operational construct, the field continues to be largely 
dominated by a narrow definition of academic language as academic vocabulary. In response to 
various researchers’ calls for more expansive definitions (Nagy & Townsend, 2012; National 
Research Council, 2010; Schleppegrell, 2004; Valdes, 2004), this study examines the potential — 
for both research and practice — of a more inclusive operationalization of an academic language 
proficiency construct. We refer to this operational construct as Core Academic Language Skills 
(CALS). We define CALS as a constellation of high-utility language skills that support academic 
reading across school content areas (e.g., knowledge of logical markers that connect ideas, such 
as nevertheless, consequently; knowledge of structures that pack dense information, such as 
nominalizations or embedded clauses; knowledge of structures for organizing analytic texts). 
Using an innovative and psychometrically robust assessment, the Core Academic Language 
Skills Instrument (CALS-I), this study explores the relationship of these cross-disciplinary 
academic language skills and reading comprehension in a socioeconomically diverse cross- 
sectional sample of students from 4**^ and 6* grade. 

Purpose / Objective / Research Question / Focus of Study: 

Two research questions guided our study: 

3. Do 4**^ and 6*-grade students’ core academic language skills —as measured by the 
CALS-I— vary by students’ grade or socioeconomic status? 

4. Controlling for socio-demographic characteristics, word recognition & decoding, and 
academic vocabulary knowledge, are 4**^ and 6*-grade students’ cross-disciplinary 
academic language skills -as measured by the CALS-I— predictive of students’ reading 
comprehension scores? 

Setting: 
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As part of the lES-funded project Catalyzing Comprehension throng Discussion and Debate 
(CCDD), the data for this study were collected as baseline data in 4*^- and 6*-grade classrooms 
that participated in the control and treatment conditions during the first year of the Word 
Generation intervention. Data were collected in seven schools in two large urban school districts 
in the Northeastern U.S. (three schools in District 1 and four schools in District 2). 

Population / Participants / Subjects: 

A total of 282 students - 85 4* graders and 197 6* graders— were included in this study. The 
sample was comprised by all the students who fulfilled two criteria: 

1 . students who participated in the pre-test data collection (Fall 201 1) of the Word 
Generation (WG) intervention in year 1 of the CCDD study; and 

2. students who had complete pre-test data for the following three assessments in study year 
1: (a) the RISE, (b) the WG Academic Vocabulary Test, and (c) the CALS Instrument. 

The overall sample was approximately balanced by students' gender and socio-economic status 
(SES), as indexed by free/reduced price lunch eligibility. A total of 149 (52%) students qualified 
for free/reduced price lunch (101 6th graders). The vast majority of students in this sample were 
classified as English proficient according to official school records. Only 4% of students were 
designated as English Language Learners (ELL). 

Intervention / Program / Practice: 

Even though all participants were taking part of the control or treatment classrooms of the WG 
intervention, the data used in this study were collected as baseline data before the intervention 
had started. Thus, the description of the intervention is irrelevant for the purposes of this paper. 

Research Design: 

Two main hypotheses are tested in this study. First, we hypothesized that CALS-I scores 
would capture individual variability in core academic language skills within and across 
grades, with students in 6**^ grade outperforming students in 4**^ grade, and students from 
more privileged SES environments outperforming their peers from less privileged 
backgrounds. Second, the main hypothesis tested by this study was whether CALS-I 
scores would be predictive of reading comprehension above and beyond the contribution 
of socio-demographic status, basic word-level reading skills, and academic vocabulary 
knowledge. To test these hypotheses, data from a socio-economically diverse sample of 
4th- and 6th-grade students was collected via group-administered assessments. Reading 
comprehension, was measured with the RISE Reading Comprehension Subtest and was 
used as outcome variable in the regression analyses. Academic language skills — 
measured by the CALS-I— were the main predictor. The covariates included basic word- 
level reading skills —measured by the RISE Word Recognition & Decoding Sub test— and 
academic vocabulary knowledge —measured using the Word Generation Academic 
Vocabulary Test. On the basis of well-documented contributions of socio-demographic 
characteristics (i.e., grade, SES), reading fluency, and academic vocabulary knowledge to 
reading comprehension, we anticipated all these covariates to be significant predictors of 
reading comprehension. Alternatively, CALS-I scores might not capture variability 
across individuals because by this age, all students have already mastered the school- 
relevant language skills tested. Moreover, given that vocabulary knowledge is positioned 
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as a critical contributor to reading comprehension, CALS-I seores might not offer any 
unique eontribution beyond the explanatory power of voeabulary knowledge. 

Data Collection and Analysis: 

Measures 

All participants completed the following group -administered assessments: 

1, Reading Inventory and Scholastic Evaluation (RISE): a 45 to 60 minute web-administered 
assessment developed by a team of researehers at ETS (Sabatini, O’Reilly, Halderman, & 
Bruee, 2014; Sabatini, Bruee, & Steinberg, 2013) to assess six constructs that underlie reading 
proficiency in the middle grades (i.e., reading comprehension, word recognition and decoding, 
vocabulary, morphological awareness, sentence proeessing, efficiency of basic reading 
comprehension). The RISE is designed for upper elementary and middle sehool students and 
provides subtest scores for eaeh reading component with the goal of informing decisions 
about literacy instruction at the district, school, and classroom levels. Two RISE sub tests are 
used in this analysis: 

(a) RISE - Reading Comprehension (RC) (22 items, 20 minutes, a = .76): RISE RC scale 
seores are used as outcome variable 

(h) RISE - Word Recognition & Decoding (WRC) (50 items, 6 minutes, a = .91): RISE 
WRC seale seores are used as a control variable. 

2, WG Academic Vocabulary test (50 items, a = .91): multiple-ehoice test that assesses 
academie words targeted by the WG intervention in grades 4 to 8. The majority of target 
words are selected from the Academic Word List (Coxhead, 2000). Each target word is 
presented in a neutral sentenee context with four responses to ehoose from: (a) a synonym 
(correct answer), an incorrect semantic associate, a phonological associate and a nonrelated 
word. The test includes 50 items. Percent eorrect scores were used for analysis (Hwang, 
Lawrence, & Snow, in preparation). 

3, Core Academic Language Skills Instrument (CALS-I) (44 items, a = .93; split 
reliability=.90): group-administered test that evaluates students' eore aeademie language skills 
in grades 4 to 8. The purpose of this test it to assess students' skills in understanding, 
producing, and reflecting upon language forms that are prevalent in aeademie texts (e.g., 
logical connectives, nominalizations). Tasks assess a range of skills through multiple choiee, 
matching, or short written responses. Two statistically equated forms with robust 
psyehometrie properties comprise the CALS-I: Eor Eorm 1 (for grades 4 to 6) and Eorm 2 (for 
grades 7 and 8). Eorm 1 was used for this study and ineluded six tasks: Connecting Ideas, 
Tracking Themes, Organizing Texts, Breaking Words, Comprehending Sentenees, Identifying 
Definitions. Prior studies have yielded robust reliability (.93 as indexed by coefficient alpha 
and .90 by split half reliability) and validity (.70 as indexed by the zero order eorrelation with 
the Gates-MaeGinitie Passage Comprehension). Raw seores were used for this analysis, but 
IRT factors scores will be generated for the final analysis. Past studies have shown raw scores 
and IRT factors scores to be highly correlated. 

Analytic Plan 

To answer RQl, we used analysis of variance with CALS-I scores as dependent variable 
and two between-subject factors: grade (two levels, grade 4 and 6) and SES (two levels, 
eligibility for free/redueed lunch=l; no eligibility=0). To answer RQ2, after examining 
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correlations between all variables, we conducted hierarchieal OLS regression analysis 
with RISE Reading Comprehension as outeome variable. First, socio-demographie 
characteristics — i.e., grade level, English language learner (EEE) designation^ SES— 
were entered as eovariates. In the second step, the word reoognition/deeoding was entered 
to the model as covariate to control for students' basie word-level skills. Then, the CALS- 
I scores were entered to examine the predictive relations between core aeademie language 
skills and reading comprehension scores above and beyond the contribution of socio- 
demographic factors (i.e., grade, SES, EEE designation) and word recognition/decoding. 

As a final step, WG aeademie vocabulary knowledge was entered to the model to explore 
the predictability of CAES-I scores even when controlling for academic vocabulary. 

Findings / Results: 

RQl: Results revealed that CAES-I scores differed significantly as a function of grade and SES. 
CALS-I scores were, on average, higher for 6* grade students (M=28.44; SD=8.5) than for 4*- 
grade students (M=20.65; SD=7.8), and also higher for students with higher SES backgrounds 
(see Table 1). 

RQ2: As the series of regression models in Table 2 illustrates, OES regression analyses revealed 
that most variables —with the only exception of grade— significantly contributed to explain the 
variability in students' reading eomprehension seores. More specifieally, as displayed in our final 
model (Model 6), performance on the CALS-I was found to be a significant predictor of students' 
reading comprehension above and beyond the eontribution of students' SES, word 
recognition/decoding, and academic vocabulary knowledge. It is interesting to note that despite 
the shared variance between aeademie vocabulary knowledge and CALS (r=.6), each of them 
offered a significant independent contribution to predict the variability in reading 
eomprehension. Notably, onee CALS-I seores and academic vocabulary knowledge^ were added 
to the model, SES was no longer a significant predictor of reading comprehension. 

Conclusions: 

While it should be of no surprise that, analogous to aeademie voeabulary knowledge, a broader 
set of general academic language skills would be predictive of reading comprehension, the 
innovation of this study resides in having identified and empirically tested an initial set of high- 
utility cross-disciplinary academic language skills associated with text comprehension. These 
results suggest that after basic word recognition/decoding skill is controlled for, the variability in 
students' core academic language skills and aeademie vocabulary capture the very pedagogically- 
relevant skills that are typieally only poorly indexed by the categorical SES variable which can 
serve only as an imperfeet proxy for identifying students in need of language and reading 
support. By speeifying a set of language skills assoeiated with reading comprehension, this study 
advances our understanding of school-relevant language skills beyond the narrow definition of 
academic vocabulary and contributes relevant information for the design of more comprehensive 
interventions that provide cognitively demanding yet linguistically accessible and productive 
instruetion. 


^ Even though the sample included only very few students designated as English Language Learners (ELL), the ELL 
designation variable was included to account for the variability in performance of this specific subgroup. The 
results of the regression analysis were comparable when ELL designation was not included in the models. 

^ In an alternative model WG Academic Vocabulary Test scores were added to the regression model before CALS-I 
scores. In this case, vocabulary scores also removed the impact of SES, even in the absence of CALS-I scores. 
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Appendix B, Tables and Figures 

Not included in page count. 


Table 1: Mean scores for all measures by grade. 


Grade 


Academic 

Language 

(CALS-I) 

Academic 

Vocabulary 

(WGAV) 

Word Reading 
Fluency 
(RISE) 

Reading 

Comprehension 

(RISE) 

4^^ grade (T otal) 

85 

20.65(7.83) 

0.65(0.18) 

361.86(27.78) 

363.07(28.33) 

grade (T otal) 

197 

28.44(8.46) 

0.60(0.18) 

362.69(28.87) 

357.22(30.31) 

Total sample 

282 

26.09(9.01) 

0.62 (0.18) 

362.80 (28.66) 

358.62 (29.60) 
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Table 2: Step-wise regression table with RISE 



Model 1 

Model 2 

Model 3 

Grade 

-2.32 (1.91) 

-2.59 (1.90) 

-2.96 (1.81) 

English 

Language 

Proficiency 

Designation 


-23.46 (8.64)*** 

-14.64(8.41) 

SES 

(Free/Reduced 

Lunch 

Eligibility) 



-18.08 (3.40) 

Word 

Recognition/ 

Decoding 

(RISE) 




Academic 

Language 

(CALS-I) 




Academic 

Vocabulary 

(WGAV) 

Observations 

282 

282 

282 

Variance 
Explained (R/ ) 

0.01 

0.03 

0.12 

Change in R/ 


0.03* 

Q 


*p<.05, **p<.01, ***p<.0001 
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Model 4 Model 5 Model 6 

-2.37(1.45) -6.10(1.61)*** -3.88(1.72)** 

-2.05(6.77) -0.59 (6.55) 1.62(6.47) 

-5.77(2.88)* -4.05(2.80) -2.76(2.78) 

0.64 0.47(0.06)** 0.37(0.07)*** 

(0.05)*** 

0.96(0.21)*** 0.67(0.22)** 

36.94(11.10)** 


282 

282 

282 

0.44 

0.48 

0.50 

0.32*** 

Q 

0.02** 
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